make_parallel for network containers #985
Conversation
| @@ -171,7 +178,8 @@ def __init__(self, | |||
| stack_size, | |||
| pooling_size=1, | |||
| dtype=torch.float32, | |||
| mode='skip'): | |||
| mode='skip', | |||
| name='TemporalPool'): | |||
There was a problem hiding this comment.
Looking at the TemporalPool example above, it seems if pool size == 2, we always ignore every other (half) of the timesteps.
Have we considered pooling every timestep, instead of pooling every pool_size timesteps?
There was a problem hiding this comment.
Is mode "avg" what you want?
|
|
||
| A parallel network has ``n`` copies of network with the same structure but | ||
| different independently initialized parameters. The parallel network can | ||
| process a batch of the data with shape [batch_size, n, ...] using ``n`` |
There was a problem hiding this comment.
Should we automatically convert data with shape [batch_size, ...] to [batch_size, n, ...]?
There was a problem hiding this comment.
As network can have sub-networks, doing this check for all of them can be wasteful. So it's intended to use make_parallel_input to do the conversion by the user.
alf/nest/utils.py
Outdated
| @@ -104,6 +105,10 @@ def _combine_flat(self, tensors): | |||
| else: | |||
| return torch.cat(tensors, dim=self._dim) | |||
|
|
|||
| def make_parallel(self, n): | |||
| dim = self._dim if self._dim < 0 else self._dim + 1 | |||
There was a problem hiding this comment.
Should add comments saying that self._dim excludes batch dim and parallel dim. The current implementation seems not to ignore the batch dim.
There was a problem hiding this comment.
Changed the behavior of NestConcat to not including batch dim.
Comments added.
* make_parallel for network containers * Address comments * Address further comments
This change supports creating parallel network for network containers. This is achieved by implementing make_parallel for various layers and transforming all the modules in the container.
With this change, it will be trivial to implement make_parallel for all kinds of networks if they are implemented using network containers.